Eyes-on-road detection

ABSTRACT

A computer includes a processor and a memory storing instructions executable by the processor to receive data indicating a gaze direction of an occupant of the vehicle; determine whether the gaze direction is in a permitted area defined by a boundary, the permitted area being in a forward direction of travel of the vehicle; laterally adjust the boundary of the permitted area based on a speed of the vehicle and based on data indicating turning of the vehicle; and control the vehicle based on whether the gaze direction is in the permitted area.

BACKGROUND

Some vehicles are equipped to perform a lane-keeping operation, i.e.,steering the vehicle to maintain a lateral position of the vehicle neara center of a lane of travel and/or away from boundaries of the lane.Typically, a computer(s) on board the vehicle uses image data from aforward-facing camera to detect the boundaries of the lane. Thecomputer(s) instructs a steering system of the vehicle to actuate toturn the wheels based on the detected boundaries of the lane.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a block diagram of an example vehicle.

FIG. 2 is a diagram of a vehicle traveling along a lane of a road.

FIG. 3 is a forward-facing view of a passenger cabin of the vehicleshowing a first example of adjusting a permitted area for a gazedirection.

FIG. 4 is a forward-facing view of the passenger cabin showing a secondexample of adjusting the permitted area.

FIG. 5 is a forward-facing view of the passenger cabin showing a thirdexample of adjusting the permitted area.

FIG. 6 is a forward-facing view of the passenger cabin showing a fourthexample of adjusting the permitted area.

FIG. 7 is a process flow diagram of an example process for adjusting thepermitted area.

DETAILED DESCRIPTION

The system described herein can control a vehicle in a lane-keepingoperation, control of the vehicle during the lane-keeping operationincluding providing outputs to a human operator related to thelane-keeping operation. Where a human operator of the vehicle may notsteer the vehicle during the lane-keeping operation, the human operatorshould be prepared to take over control of the steering. Accordingly, avehicle system can detect whether an operator's gaze direction isoutside a permitted area while the vehicle is performing a lane-keepingoperation. The permitted area is in a forward direction of travel of thevehicle. The system can provide output based on an operator's gaze beingoutside a permitted area. For example, while the vehicle is beingautonomously steered, the system can encourage the operator to keeptheir eyes on the road by outputting an alert if the gaze direction ofthe operator is outside the permitted area for a time threshold.Alternatively or additionally, the system can control the vehicle, e.g.,by outputting an alert, when the vehicle is not performing alane-keeping operation, e.g., when the vehicle is being manuallysteered.

Advantageously, the system provides an improved technique for gazemonitoring and detecting by dynamically changing the permitted area foran operator's gaze. For example, as the vehicle travels along a road, apermitted area for an operator's gaze can be dynamically changed oradjusted based on a speed of the vehicle and whether the vehicle isturning or will turn soon. For example, the permitted area can expandleftward if the vehicle is traveling over a road that curves to theleft. As the speed increases on the curve, the permitted area can expandleftward to a greater degree, permitting the operator to look fartherahead on the road while still having their gaze direction in thepermitted area. The dynamic adjustment of the permitted area can includean area in the permitted area while that area is relevant to theoperator and exclude the area when it is not relevant. Thus, systemssuch as a lane-keeping system or other driver-monitoring system that usegaze monitoring are improved, e.g., by fewer unnecessary interruptionsto the system and/or timelier and/or more targeted outputs related to adetected gaze direction.

A computer includes a processor and a memory storing instructionsexecutable by the processor to receive data indicating a gaze directionof an occupant of a vehicle, determine whether the gaze direction is ina permitted area defined by a boundary, laterally adjust the boundary ofthe permitted area based on a speed of the vehicle and based on dataindicating turning of the vehicle, and control the vehicle based onwhether the gaze direction is in the permitted area. The permitted areais in a forward direction of travel of the vehicle.

Controlling the vehicle may include outputting an alert to the occupantin response to the gaze direction being continuously outside thepermitted area for a time threshold. The alert may be at least one ofaudible, visual, and haptic.

The instructions may further include instructions to adjust the timethreshold based on the speed of the vehicle and based on the dataindicating turning of the vehicle. The time threshold may have aninverse relationship with the speed of the vehicle.

The boundary may include a first portion on a lateral side toward whichthe vehicle is turning, the first portion may be at a default positionwhen the vehicle is traveling straight, and laterally adjusting theboundary of the permitted area may include moving the first portion ofthe boundary a distance from the default position in a direction towardwhich the vehicle is turning. The distance may have a positiverelationship with a yaw rate of the vehicle.

The distance may have an inverse relationship with the speed of thevehicle.

In response to the data indicating turning of the vehicle being alane-change instruction or an active status of a turn indicator, thedistance may be a preset distance. In response to the data indicatingturning of the vehicle being the lane-change instruction or the activestatus of the turn indicator, the permitted area may include a side-viewmirror of the vehicle.

The boundary may include a second portion on a lateral side away fromwhich the vehicle is turning, and laterally adjusting the boundary ofthe permitted area may include keeping the second portion of theboundary stationary while the vehicle is turning.

The boundary may include a second portion on a lateral side away fromwhich the vehicle is turning, and laterally adjusting the boundary ofthe permitted area includes moving the second portion of the boundary inthe direction toward which the vehicle is turning.

The data indicating turning of the vehicle may include a steering-wheelangle.

The data indicating turning of the vehicle may include a lane-changeinstruction.

The data indicating turning of the vehicle may include a status of aturn indicator. The status of the turn indicator may indicate turning ofthe vehicle when the status is active up to a time threshold, and thestatus of the turn indicator may not indicate turning when the status isactive for longer than the time threshold.

The instructions may further include instructions to vertically adjustthe boundary of the permitted area based on a distance to anintersection. Vertically adjusting the boundary of the permitted areamay be based on a grade angle of a road on which the vehicle istraveling.

The instructions may further include instructions to instruct a steeringsystem of the vehicle to perform a lane-keeping operation, andcontrolling the vehicle based on whether the gaze direction is in thepermitted area may occur while performing the lane-keeping operation.

A method includes receiving data indicating a gaze direction of anoccupant of a vehicle, determining whether the gaze direction is in apermitted area defined by a boundary, laterally adjusting the boundaryof the permitted area based on a speed of the vehicle and based on dataindicating turning of the vehicle, and controlling the vehicle based onwhether the gaze direction is in the permitted area. The permitted areais in a forward direction of travel of the vehicle.

With reference to the Figures, wherein like numerals indicate like partsthroughout the several views, a computer 102 includes a processor and amemory storing instructions executable by the processor to instruct asteering system 104 of a vehicle 100 to perform a lane-keepingoperation; receive data indicating a gaze direction of an occupant ofthe vehicle 100; determine whether the gaze direction is in a permittedarea A defined by a boundary B, the permitted area A being in a forwarddirection of travel of the vehicle 100; laterally adjust the boundary Bof the permitted area A based on a speed of the vehicle 100 and based ondata indicating turning of the vehicle 100; and while performing thelane-keeping operation, control the vehicle 100 based on whether thegaze direction is in the permitted area A.

With reference to FIG. 1 , the vehicle 100 may be any passenger orcommercial automobile such as a car, a truck, a sport utility vehicle, acrossover, a van, a minivan, a taxi, a bus, etc.

The computer 102 is a microprocessor-based computing device, e.g., ageneric computing device including a processor and a memory, anelectronic controller or the like, a field-programmable gate array(FPGA), an application-specific integrated circuit (ASIC), a combinationof the foregoing, etc. Typically, a hardware description language suchas VHDL (Very High Speed Integrated Circuit Hardware DescriptionLanguage) is used in electronic design automation to describe digitaland mixed-signal systems such as FPGA and ASIC. For example, an ASIC ismanufactured based on VHDL programming provided pre-manufacturing,whereas logical components inside an FPGA may be configured based onVHDL programming, e.g., stored in a memory electrically connected to theFPGA circuit. The computer 102 can thus include a processor, a memory,etc. The memory of the computer 102 can include media for storinginstructions executable by the processor as well as for electronicallystoring data and/or databases, and/or the computer 102 can includestructures such as the foregoing by which programming is provided. Thecomputer 102 can be multiple computers coupled together.

The computer 102 may transmit and receive data through a communicationsnetwork 106 such as a controller area network (CAN) bus, Ethernet, WiFi,Local Interconnect Network (LIN), onboard diagnostics connector(OBD-II), and/or by any other wired or wireless communications network.The computer 102 may be communicatively coupled to the steering system104, sensors 108 including a speedometer 110 and a camera 112, turnindicators 114, a user interface 116, and other components via thecommunications network 106.

The steering system 104 is typically a conventional vehicle steeringsubsystem and controls the turning of the wheels. The steering system104 may be a rack-and-pinion system with electric power-assistedsteering, a steer-by-wire system, as both are known, or any othersuitable system. The steering system 104 can include an electroniccontrol unit (ECU) or the like that is in communication with andreceives input from the computer 102 and/or a human operator. The humanoperator may control the steering system 104 via, e.g., a steeringwheel.

The sensors 108 may provide data about operation of the vehicle 100, forexample, wheel speed, wheel orientation, and engine and transmissiondata (e.g., temperature, fuel consumption, etc.). The sensors 108 maydetect the location and/or orientation of the vehicle 100. For example,the sensors 108 may include global positioning system (GPS) sensors;accelerometers such as piezo-electric or microelectromechanical systems(MEMS); gyroscopes such as rate, ring laser, or fiber-optic gyroscopes;inertial measurements units (IMU); and magnetometers. The sensors 108may detect the external world, e.g., objects and/or characteristics ofsurroundings of the vehicle 100, such as other vehicles, road lanemarkings, railway crossings, traffic lights and/or signs, pedestrians,cyclists, etc. For example, the sensors 108 may include radar sensors,scanning laser range finders, light detection and ranging (LIDAR)devices, and cameras.

The sensors 108 include the speedometer 110. The speedometer 110 may beany sensor suitable for measuring the speed of the vehicle 100, forexample, as is known, a mechanical or eddy-current speedometer, or avehicle speed sensor. A vehicle speed sensor may use a magnetic fielddetector to count interruptions of a magnetic field by a toothed metaldisk disposed on a driveshaft of the vehicle 100.

The sensors 108 include at least one camera 112 aimed at the humanoperator, i.e., with a field of view encompassing the human operator.For example, the camera 112 can be located on the dashboard and orientedin a vehicle-rearward direction. The camera 112 can detectelectromagnetic radiation in some range of wavelengths. For example, thecamera 112 may detect visible light, infrared radiation, ultravioletlight, or some range of wavelengths including visible, infrared, and/orultraviolet light.

The turn indicators 114 are lamps on an exterior of the vehicle 100,e.g., on the corners of the vehicle 100, that are configured to blink tosignal an intent to turn the vehicle 100 to other vehicles. The turnindicators 114 have a status that is either active, i.e., blinking, orinactive, i.e., not blinking. The status can be represented by a binaryvalue. The status of the turn indicators 114 can be broadcast over thecommunications network 106 and is thereby available to the computer 102.

The user interface 116 presents information to and receives informationfrom an occupant of the vehicle 100, e.g., the human operator. The userinterface 116 may be located, e.g., on an instrument panel in apassenger cabin 118 of the vehicle 100, distributed over multiplelocations, and/or wherever may be readily seen, heard, or felt by theoccupant. The user interface 116 may include dials, digital readouts,screens, speakers, vibratory actuators, and so on for providinginformation to the occupant, e.g., human-machine interface (HMI)elements such as are known. The user interface 116 may include buttons,knobs, keypads, microphone, and so on for receiving information from theoccupant.

With reference to FIG. 2 , the computer 102 can be programmed to performa lane-keeping operation while traveling along a lane 120 of a road 122.The operator can activate and deactivate the lane-keeping operation,e.g., via the user interface 116. The lane-keeping operation includessteering the vehicle 100, i.e., actuating the steering system 104, tomaintain a lateral position of the vehicle 100 in the lane 120, e.g., ata center line of the lane 120 or at least a preset lateral distance awayfrom respective left and right boundaries of the lane 120. The centerline is typically an imaginary line in a longitudinal direction of thelane 120 having a same lateral distance to the respective right and leftboundaries of the lane 120. For example, the computer 102 can identifythe boundaries of the lane 120 using, e.g., an image histogram or imagesegmentation, as are known, on image data from a forward-facing cameraof the sensors 108. The computer 102 can then determine a polynomialequation, e.g., a third-degree polynomial, that predicts points on thecenter line of the lane 120. The computer 102 can determine a plannedcurvature for the path followed by the vehicle 100 using the polynomialalong with the lateral position and heading of the vehicle 100. Thecomputer 102 can determine a torque for the steering system 104 to applyby minimizing an error between the planned curvature and an actualcurvature of the vehicle 100, e.g., by using proportional integralderivative (PID) control. Finally, the computer 102 can instruct thesteering system 104 to apply the torque to turn the road wheels.

The lane-keeping operation may include a lane-change operation, e.g.,when requested by the operator. The lane-change operation is the vehicle100 moving from a current lane 120 to an adjacent lane 120 whileperforming the lane-keeping operation. While the computer 102 isperforming the lane-keeping operation, the operator can input alane-change instruction, e.g., via the user interface 116. The computer102 can receive data from the sensors 108, e.g., radars positioned onsides of the vehicle 100, to determine whether the adjacent lane 120 isclear. If so, then the computer 102 can determine a polynomial equationdescribing motion from the current lane 120 to the adjacent lane 120 andproceed as described above for the lane-keeping operation.

With reference to FIGS. 3-6 , the permitted area A is a range ofpossible directions of the gaze of the occupant. As described below withrespect to a process 700, the computer 102 uses whether the gazedirection of the operator is in the permitted area A to control thevehicle 100. The range of the permitted area A can be described in twoangular dimensions, e.g., the operator tilting their head and/or eyes upand down, and the operator turning their head and/or eyes left andright. The range of the permitted area A can equivalently be describedaccording to locations on a surface of a passenger cabin 118 of thevehicle 100, e.g., a left half of a windshield 124, a rear-view mirror,etc. toward which the gaze direction of the operator can be directed.The permitted area A has a default position when the vehicle 100 istraveling straight on a level grade. The default position of thepermitted area A can be chosen to encompass the gaze directions at whichthe operator is looking at the road 122 when the road is straight andlevel based on empirical testing, i.e., obtaining measurements of gazedirections of operators in vehicles. For example, the default positionof the permitted area A could be a portion of a left half of thewindshield 124 extending from the bottom of the windshield 124 partiallytoward the top of the windshield 124. As described below with respect tothe process 700, the permitted area A is adjustable from the defaultposition in various circumstances. The permitted area A is in a forwarddirection of travel of the vehicle 100, both when in the defaultposition and after adjustments.

The permitted area A is defined by the boundary B. The adjustments ofthe permitted area A can be described in terms of moving portions P1,P2, P3, P4 of the boundary B, with prime (′) or double prime (″)indicating post-adjustment positions in the Figures. As describedfurther below, FIGS. 3-6 illustrate examples of how the permitted area Acan change based on view speed and turning. For example, when thevehicle 100 is turning left, a left lateral portion of the boundary Bcan move leftward, as shown in the examples of FIGS. 3 and 4 .Typically, the permitted area A is a rectangle, and the portions P1, P2,P3, P4 described variously with respect to FIGS. 3-6 are line segmentsdefining one side of the rectangle.

The boundary B of the permitted area A is adjusted based on the speed ofthe vehicle 100, data indicating turning of the vehicle 100, distance toan intersection, and/or grade angle of the road 122. The data indicatingturning of the vehicle 100 can include data indicating that the vehicle100 is currently turning or will turn soon, e.g., will turn in at most atime threshold before which the computer 102 outputs an alert, asdescribed below with respect to the process 700. The data indicatingturning of the vehicle 100 can include the status of the turn indicators114, a steering-wheel angle, a yaw rate of the vehicle 100, thelane-change instruction for the lane-change operation, GPS data, mapdata, image data from a forward-facing camera of the sensors 108, etc.For example, the sensors 108 can include, e.g., a torque sensor or aposition sensor in the steering system 104, e.g., on a steering column,for reporting the steering-wheel angle. For another example, thecomputer 102 can determine a current curvature and/or an upcomingcurvature of a road on which the vehicle 100 is traveling based on themap data or based on an image histogram or image segmentation of theimage data, as described above with respect to the lane-keepingoperation.

The adjustment of the boundary B can be lateral and/or vertical. If thevehicle 100 is traveling straight and/or will continue to travelstraight, the boundary B is laterally maintained at the defaultposition. If the vehicle 100 is on a level grade and is far from anintersection, the boundary B is vertically maintained at the defaultposition. As these conditions are not met, the computer 102 adjusts theboundary B laterally and/or vertically. FIGS. 3 and 4 are two examplesof laterally adjusting the boundary B, and FIGS. 5 and 6 are twoexamples of additional ways to adjust the boundary B that can each beincluded with either of the examples of Figures A and B. For example,the computer 102 can laterally adjust the boundary B of the permittedarea A based on a speed of the vehicle 100 and based on data indicatingturning of the vehicle 100, as in the examples of FIG. 3 or 4 ; thecomputer 102 can laterally adjust the permitted area A by a presetdistance based on the data indicating turning of the vehicle 100 beingan active status of the turn indicators 114 or a lane-changeinstruction, as in the example of FIG. 5 ; and/or the computer 102 canvertically adjust the permitted area A based on a distance of thevehicle 100 to an intersection, the speed of the vehicle 100, and/or agrade angle of a road 122 on which the vehicle 100 is traveling, as inthe example of FIG. 6 . These examples will now be described in turn.

With respect to the examples of FIGS. 3 and 4 , the computer 102 canlaterally adjust the boundary B of the permitted area A by moving afirst portion P1 of the boundary B a distance D from the defaultposition. In the example of FIGS. 3 and 4 , the boundary B includes thefirst portion P1 on a lateral side toward which the vehicle 100 isturning, e.g., a left side if the vehicle 100 is turning left, as seenin FIG. 3 , or a right side if the vehicle 100 is turning right, as seenin FIG. 4 .

The distance D is measured from the default position of the firstportion P1 of the boundary B extending in the direction that the vehicle100 is turning. The distance D is a function of the data indicating thatthe vehicle 100 is turning, e.g., the yaw rate r, and of the speed V ofthe vehicle 100, e.g., D=D(r, V). Other data indicating turning, asdescribed above, could be used in addition to or in lieu of the yaw rater. For example, the function for the distance D can stored in the memoryof the computer 102 as a lookup table. When the yaw rate r is zero,i.e., when the vehicle 100 is traveling straight, the first portion P1is at the default position, i.e., D(0, V)=0. The distance D has apositive relationship with the yaw rate r of the vehicle 100; i.e., asthe yaw rate r increases, the distance D increases. In other words, thefirst portion P1 of the boundary B shifts farther when the vehicle 100turns more sharply. For example, as shown in FIG. 3 , the first portionP1 can move a short distance D to a new position P1′ when the yaw rate ris small or a large distance D′ to a new position P1″ when the yaw rater is large. The distance D has an inverse relationship with the speed Vof the vehicle 100; i.e., as the speed V increases, the distance Ddecreases. In other words, as the vehicle 100 travels faster, the firstportion P1 of the boundary B shifts to a lesser degree when the vehicle100 turns. The function for the distance D, e.g., values for the lookuptable for determining the distance D, can be chosen to encompass thegaze directions at which the operator is looking at the road 122 adistance ahead corresponding to a reaction time of the operator when theroad curves consistent with various yaw rates r and the vehicle 100 istraveling at various speeds V, based on empirical or experimentalmeasurements of operators in vehicles.

With respect specifically to the example of FIG. 3 , the boundary Bincludes a second portion P2 on a lateral side away from a direction inwhich the vehicle 100 is turning. The second portion P2 is on theopposite lateral side of the permitted area A as the first portion P1.Laterally adjusting the boundary B can include keeping the secondportion P2 of the boundary B stationary at the default position whilethe vehicle 100 is turning, i.e., while the first portion P1 of theboundary B is moved a distance D from the default position. In otherwords, as the vehicle 100 turns, the permitted area A widens by thedistance D. This example can avoid behavior that is unexpected by theoperator such as providing an alert when the operator is lookingstraight forward.

Alternatively, with respect specifically to the example of FIG. 4 ,laterally adjusting the boundary B can include moving the second portionP2 of the boundary B in the direction toward which the vehicle 100 isturning, e.g., to the left when the vehicle 100 is turning left. Forexample, when the first portion P1 moves by the distance D in thedirection in which the vehicle 100 is turning to a new position P1′, thesecond portion P2 can move by the distance D in the same direction to anew position P2′. In other words, as the vehicle 100 turns, thepermitted area A has a constant width. This example can encourage theoperator to turn their gaze in the direction in which the vehicle 100 isturning. A manufacturer of the vehicle 100 can choose which one of theexamples of FIGS. 3 and 4 to install by testing a likelihood of theoperator continuing to look forward during a turn, how operators respondto receiving alerts when looking forward, and customer preferences aboutsuch alerts.

With respect to the example of FIG. 5 , in response to the dataindicating turning of the vehicle 100 being the lane-change instructionor the active status of the turn indicators 114, the computer 102laterally adjusts the boundary B of the permitted area A by moving thefirst portion P1 of the boundary B the distance D from the defaultposition to a new position P1′, and the distance D is a preset distance,i.e., D=D₀. The preset distance Do is a value stored in memory. Thepreset distance Do can be chosen to encompass a view of the adjacentlane 120 into which the vehicle 100 is moving. The distance D can be setto different preset distances depending on whether the vehicle 100 ischanging lanes 120 to the left or to the right, i.e., D_(0L)≠D_(0R).Additionally, the permitted area A can include a side-view mirror 126,e.g., the side-view mirror 126 on a side of the vehicle 100 toward whichthe vehicle 100 is turning. For example, the permitted area A can benoncontiguous, with a portion defined by a second boundary B′encompassing the side-view mirror 126 spaced from a main portion of thepermitted area A. The portion of the permitted area A defined by thesecond boundary B′ can be rectangular. In the other examples, thepermitted area A is contiguous.

With respect to the example of FIG. 6 , the computer 102 can verticallyadjust the permitted area A based on the distance of the vehicle 100 toan intersection, the speed of the vehicle 100, and/or a grade angle of aroad 122 on which the vehicle 100 is traveling. The intersection can belimited to intersections having stoplights, i.e., intersections at whichthe operator will need to look upward to see a traffic signal. Thecomputer 102 can vertically adjust the permitted area A by moving athird portion P3 of the boundary B a distance D_(z) from the defaultposition. The boundary B includes the third portion P3 on a top side ofthe permitted area A. The distance D_(z) is measured from the defaultposition of the third portion P3 of the boundary B. The distance D_(z)is a function of the distance X to an intersection, the speed V of thevehicle 100, and a grade angle θ of the road 122, e.g., D_(z)=D_(z)(X,V, θ). When the distance X from the vehicle 100 to the next intersectionis greater than a threshold distance X₀, the distance D_(z) is zero,i.e., D_(z)(X>X₀, V, θ)=0. The distance D_(z) has an inverserelationship with the distance X, i.e., as the distance X decreases, thedistance D_(z) increases. In other words, as the vehicle 100 nears theintersection, the third portion P3 moves farther up. The distance D_(z)has a positive relationship with the grade angle θ, i.e., as the gradeangle θ increases, the distance D_(z) increases. Additionally oralternatively, the distance D_(z) can be a function of a rate of changeof the grade dθ/dx, e.g., a transition from flat to an upward ordownward grade, and/or a function of a travel distance to a grade havingat least a specific steepness, e.g., to a beginning of a hill ordescent. The boundary B includes a fourth portion P4 on a bottom side ofthe permitted area A. Vertically adjusting the permitted area A caninclude keeping the fourth portion P4 of the boundary B stationary atthe default position while the third portion P3 is moved the distanceD_(z) from the default position, i.e., the permitted area A growingtaller. Alternatively, as shown in FIG. 6 , when the third portion P3 ofthe boundary B moves by the distance D_(z) to a new position P3′,vertically adjusting the permitted area A can include moving the fourthportion P4 of the boundary B by the distance D_(z) to a new positionP4′, i.e., the permitted area A has a constant height.

For another example, the computer 102 can laterally adjust the boundaryB of the permitted area A by moving the first portion P1 or secondportion P2 of the boundary B the distance D from the default position inresponse to detecting a nonvehicle road user, e.g., a pedestrian orcyclist. The distance D can be chosen based on a position of thenonvehicle road user so that the permitted area A includes the gazedirection of the operator being directed at the nonvehicle road user.

FIG. 7 is a process flow diagram illustrating an exemplary process 700for adjusting the permitted area A. The memory of the computer 102stores executable instructions for performing the steps of the process700 and/or programming can be implemented in structures such asmentioned above. The process 700 is started by the computer 102receiving an input to that a trigger condition has occurred, i.e., acondition during which a gaze direction is monitored, e.g., thelane-keeping operation is activated, the vehicle 100 is shifted into aforward gear, the vehicle 100 is traveling with a positive speed, etc.

As a general overview of the process 700, while the trigger conditioncontinues, the computer 102 receives data, performs the lane-keepingoperation if applicable, adjusts the boundary B of the permitted area A,and adjusts time thresholds for providing an alert. The boundary B andthe time thresholds can be adjusted based on the speed of the vehicle100, data indicating turning of the vehicle 100, distance to anintersection, and/or grade angle of the road 122. The computer 102controls the vehicle 100 based on whether the gaze direction is outsidethe adjusted permitted area A. For example, when the gaze direction ofthe operator is outside the permitted area A, the computer 102increments a counter and provides an alert to the operator whenever thecounter reaches one of a plurality of time thresholds. The computer 102resets the counter when the gaze direction is within the permitted areaA or, if an alert has been outputted, within the permitted areacontinuously for a resumption time threshold. The process 700 ends whenthe computer 102 receives an input that the trigger condition is nolonger occurring.

The process 700 begins in a block 705, in which the computer 102receives and processes data from the sensors 108 and data of the statusof the turn indicators 114. The data from the sensors 108 includes datafrom a forward-facing camera of the sensors 108 of an area in front ofthe vehicle 100, to which the computer 102 can apply the image histogramor image segmentation discussed above. The data includes the speed ofthe vehicle 100 from the speedometer 110. The data includes the dataindicating turning of the vehicle 100, e.g., the status of the turnindicators 114, a steering-wheel angle, a yaw rate of the vehicle 100,the lane-change instruction for the lane-change operation, GPS data, mapdata, image data from a forward-facing camera of the sensors 108, etc.The data includes image data from the camera 112 that is aimed at theoperator of the vehicle 100. The computer 102 can process the image datato determine the gaze direction of the operator. For example, thecomputer 102 can detect the face in the image data from the camera 112,e.g., by using any suitable facial-detection technique, e.g.,knowledge-based techniques such as a multiresolution rule-based method;feature-invariant techniques such as grouping of edges, space gray-leveldependence matrix, or mixture of Gaussian; template-matching techniquessuch as shape template or active shape model; or appearance-basedtechniques such as eigenface decomposition and clustering, Gaussiandistribution and multilayer perceptron, neural network, support vectormachine with polynomial kernel, a naive Bayes classifier with jointstatistics of local appearance and position, higher order statisticswith hidden Markov model, or Kullback relative information. Then thecomputer 102 can use outputs produced as a byproduct of the facialdetection that indicate the gaze direction.

Next, in a block 710, if the trigger condition was activating thelane-keeping operation, the computer 102 actuates the steering system104 to perform the lane-keeping operation as described above.

Next, in a block 715, the computer 102 adjusts the boundary B of thepermitted area A based on the speed of the vehicle 100, data indicatingturning of the vehicle 100, distance to an intersection, and/or gradeangle of the road 122, as described above with respect to FIGS. 3-6 .

Next, in a block 720, the computer 102 adjusts the time threshold(s)T_(i) based on the speed V of the vehicle 100 and based on the dataindicating turning of the vehicle 100. A time threshold T_(i) is alength of time that the operator can have their gaze direction outsidethe permitted area A before the computer 102 outputs an alert. Thecomputer 102 can be programmed to have multiple time thresholds T_(i),e.g., a first time threshold T₁ for a first alert, a second timethreshold T₂ for a second alert, etc. The time thresholds T_(i) can havedefault values, e.g., T_(1def)=5 seconds for a first alert, T_(2def)=8seconds for a second alert (i.e., 3 seconds after the first alert), andT_(3def)=13 seconds for a third alert (i.e., 5 seconds after the secondalert). The time thresholds T_(i), are a function of the data indicatingthat the vehicle 100 is turning, e.g., the yaw rate r, and of the speedV of the vehicle 100, e.g., T_(i)(r, V) for i=1, 2, 3. The timethresholds T_(i) can have an inverse relationship with the yaw rate r ofthe vehicle 100; i.e., as the yaw rate r increases, the time thresholdsT_(i) decrease. The time thresholds T_(i) can have an inverserelationship with the speed V of the vehicle 100; i.e., as the speed Vincreases, the time thresholds T_(i) decrease.

Next, in a decision block 725, the computer 102 determines whether thegaze direction of the operator is in the permitted area A. The computer102 compares the gaze direction from the block 705 with the permittedarea A after the adjustments of the block 715. If an alert has beenoutputted in a block 745 below for the gaze direction being outside theboundary B of the permitted area A for longer than one of the timethresholds T_(i), the computer 102 can determine whether the gazedirection of the operator is continuously in the permitted area for atleast a resumption time threshold. The resumption time threshold can bechosen to be sufficiently long for the operator to understand a road andtraffic situation, as based on experimental testing of operators. If thealert has been outputted, the computer 102 may additionally require thatthe operator provide an input e.g., place their hands on the steeringwheel, tap a brake pedal, provide an input to the user interface 116,etc. If the gaze direction is within the boundary B of the permittedarea A (for at least the resumption time threshold if applicable), theprocess 700 proceeds to a block 730. If the gaze direction is outsidethe boundary B of the permitted area A (for at least the resumption timethreshold if applicable), the process 700 proceeds to a block 735.

In the block 730, the computer 102 resets a counter to zero. The countertracks how long the gaze direction of the operator is outside thepermitted area A. Whenever the gaze direction of the operator returns tothe permitted area A as determined in the decision block 725, thecounter resets to zero in this block 730. After the block 730, theprocess 700 proceeds to a decision block 750.

In the block 735, the computer 102 increments the counter. In otherwords, the counter counts upward since the last time the counter wasreset in the block 730. The counter thus tracks a time that the gazedirection is continuously outside the permitted area A, i.e., a timethat the gaze direction is outside the permitted area A withoutinterruption.

Next, in a decision block 740, the computer 102 determines whether thecounter is at one of the time thresholds, i.e., whether a value t of thecounter equals, e.g., T₁ or T₂ or T₃. If the value t of the counter isat one of the time thresholds T_(i) the process 700 proceeds to theblock 745. If the value t of the counter is not equal to any of the timethresholds T_(i), the process 700 proceeds to the decision block 750.

In the block 745, the computer 102 outputs an alert to the operator. Thealert is at least one of audible, visual, or haptic. For example, thecomputer 102 can instruct the user interface 116 to display a messageand/or symbol in the instrument cluster and/or to sound a beep or chime.The alert could additionally be haptic, e.g., vibrating a seat in whichthe operator is sitting. After the block 745, the process 700 proceedsto the decision block 750.

In the decision block 750, the computer 102 determines whether thetrigger condition is no longer occurring. For example, if the triggercondition was the lane-keeping operation, the trigger condition ceaseswhen the operator provides an input via the user interface 116 todeactivate the lane-keeping operation. If the trigger condition is thevehicle 100 being in a forward gear, the trigger condition ceases whenthe vehicle 100 is put into a nonforward gear, e.g., park, neutral, orreverse. If the trigger condition is the vehicle 100 traveling forward,the trigger condition ceases when the vehicle 100 is stopped ortraveling in reverse. If the trigger condition is still occurring, theprocess 700 returns to the block 705 to continue receiving the data. Ifthe trigger condition has ceased, the process 700 ends.

In general, the computing systems and/or devices described may employany of a number of computer operating systems, including, but by nomeans limited to, versions and/or varieties of the Ford Sync®application, AppLink/Smart Device Link middleware, the MicrosoftAutomotive® operating system, the Microsoft Windows® operating system,the Unix operating system (e.g., the Solaris® operating systemdistributed by Oracle Corporation of Redwood Shores, Calif.), the AIXUNIX operating system distributed by International Business Machines ofArmonk, N.Y., the Linux operating system, the Mac OSX and iOS operatingsystems distributed by Apple Inc. of Cupertino, Calif., the BlackBerryOS distributed by Blackberry, Ltd. of Waterloo, Canada, and the Androidoperating system developed by Google, Inc. and the Open HandsetAlliance, or the QNX® CAR Platform for Infotainment offered by QNXSoftware Systems. Examples of computing devices include, withoutlimitation, an on-board vehicle computer, a computer workstation, aserver, a desktop, notebook, laptop, or handheld computer, or some othercomputing system and/or device.

Computing devices generally include computer-executable instructions,where the instructions may be executable by one or more computingdevices such as those listed above. Computer executable instructions maybe compiled or interpreted from computer programs created using avariety of programming languages and/or technologies, including, withoutlimitation, and either alone or in combination, Java™, C, C++, Matlab,Simulink, Stateflow, Visual Basic, Java Script, Python, Perl, HTML, etc.Some of these applications may be compiled and executed on a virtualmachine, such as the Java Virtual Machine, the Dalvik virtual machine,or the like. In general, a processor (e.g., a microprocessor) receivesinstructions, e.g., from a memory, a computer readable medium, etc., andexecutes these instructions, thereby performing one or more processes,including one or more of the processes described herein. Suchinstructions and other data may be stored and transmitted using avariety of computer readable media. A file in a computing device isgenerally a collection of data stored on a computer readable medium,such as a storage medium, a random access memory, etc.

A computer-readable medium (also referred to as a processor-readablemedium) includes any non-transitory (e.g., tangible) medium thatparticipates in providing data (e.g., instructions) that may be read bya computer (e.g., by a processor of a computer). Such a medium may takemany forms, including, but not limited to, non-volatile media andvolatile media. Non-volatile media may include, for example, optical ormagnetic disks and other persistent memory. Volatile media may include,for example, dynamic random access memory (DRAM), which typicallyconstitutes a main memory. Such instructions may be transmitted by oneor more transmission media, including coaxial cables, copper wire andfiber optics, including the wires that comprise a system bus coupled toa processor of a ECU. Common forms of computer-readable media include,for example, a floppy disk, a flexible disk, hard disk, magnetic tape,any other magnetic medium, a CD-ROM, DVD, any other optical medium,punch cards, paper tape, any other physical medium with patterns ofholes, a RAM, a PROM, an EPROM, a FLASH-EEPROM, any other memory chip orcartridge, or any other medium from which a computer can read.

Databases, data repositories or other data stores described herein mayinclude various kinds of mechanisms for storing, accessing, andretrieving various kinds of data, including a hierarchical database, aset of files in a file system, an application database in a proprietaryformat, a relational database management system (RDBMS), a nonrelationaldatabase (NoSQL), a graph database (GDB), etc. Each such data store isgenerally included within a computing device employing a computeroperating system such as one of those mentioned above, and are accessedvia a network in any one or more of a variety of manners. A file systemmay be accessible from a computer operating system, and may includefiles stored in various formats. An RDBMS generally employs theStructured Query Language (SQL) in addition to a language for creating,storing, editing, and executing stored procedures, such as the PL/SQLlanguage mentioned above.

In some examples, system elements may be implemented ascomputer-readable instructions (e.g., software) on one or more computingdevices (e.g., servers, personal computers, etc.), stored on computerreadable media associated therewith (e.g., disks, memories, etc.). Acomputer program product may comprise such instructions stored oncomputer readable media for carrying out the functions described herein.

In the drawings, the same reference numbers indicate the same elements.Further, some or all of these elements could be changed. With regard tothe media, processes, systems, methods, heuristics, etc. describedherein, it should be understood that, although the steps of suchprocesses, etc. have been described as occurring according to a certainordered sequence, such processes could be practiced with the describedsteps performed in an order other than the order described herein. Itfurther should be understood that certain steps could be performedsimultaneously, that other steps could be added, or that certain stepsdescribed herein could be omitted.

All terms used in the claims are intended to be given their plain andordinary meanings as understood by those skilled in the art unless anexplicit indication to the contrary in made herein. In particular, useof the singular articles such as “a,” “the,” “said,” etc. should be readto recite one or more of the indicated elements unless a claim recitesan explicit limitation to the contrary. Use of “in response to” and“upon determining” indicates a causal relationship, not merely atemporal relationship. The adjectives “first,” “second,” “third,” and“fourth” are used throughout this document as identifiers and are notintended to signify importance, order, or quantity.

The disclosure has been described in an illustrative manner, and it isto be understood that the terminology which has been used is intended tobe in the nature of words of description rather than of limitation. Manymodifications and variations of the present disclosure are possible inlight of the above teachings, and the disclosure may be practicedotherwise than as specifically described.

What is claimed is:
 1. A computer comprising a processor and a memorystoring instructions executable by the processor to: receive dataindicating a gaze direction of an occupant of a vehicle; determinewhether the gaze direction is in a permitted area defined by a boundary,the permitted area being in a forward direction of travel of thevehicle; laterally adjust the boundary of the permitted area based on alongitudinal speed of the vehicle and based on data indicating turningof the vehicle; vertically adjust the boundary of the permitted areabased on a distance to an intersection; and control the vehicle based onwhether the gaze direction is in the permitted area.
 2. The computer ofclaim 1, wherein controlling the vehicle includes outputting an alert tothe occupant in response to the gaze direction being continuouslyoutside the permitted area for a time threshold.
 3. The computer ofclaim 2, wherein the alert is at least one of audible, visual, andhaptic.
 4. The computer of claim 2, wherein the instructions furtherinclude instructions to adjust the time threshold based on thelongitudinal speed of the vehicle and based on the data indicatingturning of the vehicle.
 5. The computer of claim 4, wherein the timethreshold has an inverse relationship with the longitudinal speed of thevehicle.
 6. The computer of claim 1, wherein the boundary includes afirst portion on a lateral side toward which the vehicle is turning; thefirst portion is at a default position when the vehicle is travelingstraight; and laterally adjusting the boundary of the permitted areaincludes moving the first portion of the boundary a second distance fromthe default position in a direction toward which the vehicle is turning.7. The computer of claim 6, wherein the second distance has a positiverelationship with a yaw rate of the vehicle.
 8. The computer of claim 6,wherein the second distance has an inverse relationship with thelongitudinal speed of the vehicle.
 9. The computer of claim 6, wherein,in response to the data indicating turning of the vehicle being alane-change instruction or an active status of a turn indicator, thesecond distance is a preset distance.
 10. The computer of claim 6,wherein the boundary includes a second portion on a lateral side awayfrom which the vehicle is turning, and laterally adjusting the boundaryof the permitted area includes keeping the second portion of theboundary stationary while the vehicle is turning.
 11. The computer ofclaim 6, wherein the boundary includes a second portion on a lateralside away from which the vehicle is turning, and laterally adjusting theboundary of the permitted area includes moving the second portion of theboundary in the direction toward which the vehicle is turning.
 12. Thecomputer of claim 1, wherein the data indicating turning of the vehicleinclude a steering-wheel angle.
 13. The computer of claim 1, wherein thedata indicating turning of the vehicle include a lane-changeinstruction.
 14. The computer of claim 1, wherein the data indicatingturning of the vehicle include a status of a turn indicator.
 15. Thecomputer of claim 14, wherein the status of the turn indicator indicatesturning of the vehicle when the status is active up to a time threshold,and the status of the turn indicator does not indicate turning when thestatus is active for longer than the time threshold.
 16. The computer ofclaim 1, wherein the instructions further include instructions tovertically adjust the boundary of the permitted area based on a gradeangle of a road on which the vehicle is traveling.
 17. The computer ofclaim 1, wherein the instructions further include instructions toinstruct a steering system of the vehicle to perform a lane-keepingoperation, and controlling the vehicle based on whether the gazedirection is in the permitted area occurs while performing thelane-keeping operation.
 18. A method comprising: receiving dataindicating a gaze direction of an occupant of a vehicle; determiningwhether the gaze direction is in a permitted area defined by a boundary,the permitted area being in a forward direction of travel of thevehicle; laterally adjusting the boundary of the permitted area based ona longitudinal speed of the vehicle and based on data indicating turningof the vehicle; vertically adjusting the boundary of the permitted areabased on a distance to an intersection; and controlling the vehiclebased on whether the gaze direction is in the permitted area.
 19. Thecomputer of claim 1, wherein the instructions further includeinstructions to receive data indicating the longitudinal speed from aspeedometer of the vehicle.